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A METHOD AND SYSTEM FOR EVALUATING OPTIONS BASED 
ON ONE OR MORE RATINGS ALONG ONE OR MORE 

DIMENSIONS 



Background of the Invention 



[0001] The present invention relates to a method and system for evaluating 

options, and to a computer readable medium for programming the system in accordance 
with the method. More particularly it relates to evaluating options which have been rated 
along one or more subjective dimensions. (By "subjective dimension", sometimes 
hereinafter "dimension", herein is meant a characteristic of an option which can only be 
rated as a matter of opinion, e.g. style, or imprecisely estimated, e.g. useful life or resale 
value; so that the judgment and skill of the parties rating the options are important in 
evaluating the options.) 

[0002] Often people must choose between a number of options which, in at least 

some respects, cannot be compared objectively. That is, the choice among the options is 
based, at least in part, on subjective dimensions of the options. In such cases it is 
desirable to make evaluations that properly account for all available information. 
Existing sources of such information include; periodicals published by third parties, e.g. 
"Consumer Reports", files of vendor supplied information, collected opinions of previous 
users, etc. Such sources can be maintained as local hard copy files, for scanned or 
manual input, or as electronic databases, maintained either on a user's system or on a 
network such as the Internet or a local network. (Hereinafter all such information sources 
are referred to as "type 1 databases". More precisely, by "type 1 databases" herein is 
meant any source of such information which can be accessed by the system of the present 
invention, either as digital files or by manual or scanned input, and which rate at least one 
option of interest to a user along at least one dimension of interest to the user.) 
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[0003] While useful, such existing sources leave the user dependent upon the 

objectivity and reliability of their authors and editors. Furthermore, they do not provide 
the user with a mechanism for combining multiple sources. 

[0004] Thus it is an object of the subject invention to evaluate database ratings of 

a plurality of options along one or more subjective dimensions, and to combine ratings 
from a plurality of databases into an overall rating. 

Brief Summary of the Invention 

[0005] The above object is achieved and the disadvantages of the prior art are 

overcome in accordance with the subject invention by a method and system for selecting 
and accessing type 1 databases, DBS, each of the selected databases DB 1 * including at 
least one option rating, ORi(x,n), for one of the options, x, with respect to a dimension n, 
where the option x can differ among the selected databases; and selecting and accessing 
type 2 databases DB 2 j, each of the type 2 databases DB 2 j including at least one database 
rating DRj(i) for at least one of the databases DB^. Weights, Wi are associated with the 
databases DB 1 *, the weights Wj being calculated as a function of the database ratings 
DRj(i); and an overall rating R(m,n) is calculated for an option m with respect to the 
dimension n as a function of the weights Wi and option ratings ORi(m,n). The 
calculation is repeated for each remaining one of the options for which there exists at 
least one option rating with respect to the dimension n; and a list of the options and 
associated overall ratings with respect to dimension n is generated. 
[0006] In accordance with one aspect of the subject invention the function of the 

weights Wi and the option ratings ORi(m,n) is: 

R(m,n) = Zi(Wi-Norm(ORi(m,n))/ £iW i; 

where Norm(ORi(m,n)) is a normalization of the option ratings ORi(m,n), and the 
summation ranges over all of the type 1 databases DB^ for which the option ratings 
ORi(m,n) are defined. 
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[0007] In accordance with another aspect of the subject invention the option 

ratings ORi(m,n) are normalized with respect to a maximum rating ORi(max) and a 
minimum satisfactory rating ORi(sat) for each of the selected type 1 databases DB\. 
[0008] In accordance with another aspect of the subject invention , if the option 

rating OR{(m,n) is less than the minimum satisfactory rating, ORi(sat), the normalization, 
Norm(ORi(m,n)) is set equal to a predetermined value; the predetermined value being 
less than the normalized minimum satisfactory value. 

[0009] In accordance with another aspect of the subject invention the function of 

the database ratings DRj(i) is: 

Wi = £j(MWj-Norm(DRj(i))/ XjMWj; 

where Norm(DRj(i)) is a normalization of the database ratings DRj(i), and the summation 
ranges over all of the type 2 databases DB 2 j for which the option ratings DRj(i) are 
defined, and the MWj are master weights associated with the type 2 databases DB 2 j. 
[0010] In accordance with another aspect of the subject invention the database 

ratings DR j are normalized with respect to a maximum rating DRj(max) and a minimum 
satisfactory rating DRj(sat) for each of the selected type 2 databases DB 2 j. 
[0011] In accordance with still another aspect of the subject invention the master 

weights MW, are adjusted based on a user's evaluation of the list. 
[0012] In accordance with another aspect of the subject invention adjustment of 

the master weights MWj includes a user identifying a selected choice m'; and calculating 
a partial derivative P(MWj') = 3F(m ? ,n)/3MWj ? ; where F(m ? ,n) is the deviation of option 
rating R(m',n) from the mean rating, ZmR(m,n)/M as a function of master weights MWj, 
where M is the total number of options. MWj' is then set equal to MWj'O + aP(MWj')), 
where a is a small positive constant, and the process is repeated for all remaining master 
weights MWj. 

[0013] In accordance with still another aspect of the subject invention the options 

are rated with respect to a plurality of dimensions. 
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[0014] Other objects and advantages of the subject invention will be apparent to 

those skilled in the art from consideration of the detailed description set forth below and 
the attached drawings. 

Brief Description of the Drawings 

[0015] Figure 1 shows a schematic block diagram of a system in accordance with 

the present invention and an associated network and databases. 

[0016] Figures 2A and 2B show a flow diagram of the operation of the system of 

Figure 1 in accordance with the present invention. 

[0017] Figure 3 shows a flow diagram of the operation of the system of Figure 1 

in accordance with one aspect of the present invention. 

Detailed Description of Preferred Embodiments of the Invention 

[0018] Figure 1 shows system 10 in accordance with the present invention. 

System 10 communicates with type 1 databases DB\ through DB 1 ! and type 2 Databases 
DB i through DB j through network 20 in any convenient manner. Network 20 can be a 
network such as the Internet, or a local area network, or a public switched network, or 
any other convenient mechanism for communicating with a plurality of databases which 
can be located in various places and maintained by various parties. In other embodiments 
of the present invention, some, or all, of databases DB 1 and DB 2 can be maintained 
locally on system 10. 

[0019] System 10 includes data processing system 12, which is preferably a 

conventional digital computer, input/out system 14, through which a system user interacts 
with system 12, and disk reader 16 for reading portable magnetic disk 18. Input/output 
system can include a printer or visual display, or any other convenient mechanism for 
output of lists of options with associate ratings in accordance with the present invention. 
In other embodiments of the present invention input /output system 14 can include a 
keyboard or scanner for input of data from databases maintained as hard copy. 
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[0020] In the embodiment of Figure 1 system 12 is programmed to carry out the 

method of the present invention by instructions provided by portable magnetic disk 1 8 
and disk drive 16. In other embodiments of the present invention any other convenient 
computer readable medium can be used to provide instructions to system 12. The term 
"computer-readable medium" as used herein refers to any medium that participates in 
providing instructions to system 1 2 for execution. Such a medium may take many forms, 
including but not limited to, non-volatile media, volatile media, and transmission media. 
Non-volatile media include, for example, optical or magnetic disks. Volatile media 
include dynamic memory. Transmission media include coaxial cables, copper wire and 
fiber optics. Transmission media can also take the form of acoustic or electromagnetic 
waves, such as those generated during radio frequency (RF) and infrared (IR) data 
communications. Common forms of computer-readable media include, for example, a 
floppy disk, a flexible disk, hard disk, magnetic tape, any other magnetic medium, a CD- 
ROM, DVD, any other optical medium, punch cards, paper tape, any other physical 
medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, any 
other memory chip or cartridge, a carrier wave as described hereinafter, or any other 
medium from which a computer can read. 

[0021] Type 1 databases DB\ contain ratings for various types of options. For 

example databases DB 1 ! can contain ratings for consumer products, industrial products, 
professional services such as doctors, accountants, etc., institutions such as hospitals or 
schools, and in general options for any choice or situation with which a user may be 
faced. Each option is rated along one or more dimensions, i.e. characteristics of interest,. 
For example, consumer products can be rated for value, durability, service/warranty, 
overall quality, and style. Databases DB^ can also report objective information such as 
price. Note that every database DB^ need not contain ratings for every option, nor rate 
every option along every dimension. 

[0022] Type 2 databases DB j contain ratings of the relative quality of type 1 

databases DB 1 ]. Again note that each database DB l i need not be rated by every database 
DB 2 j. Preferably such ratings will reflect the experiences of other users with various type 
1 databases DB*i and the value placed on them but they can also be based on more or less 
abstract evaluations of databases DBS by third parties. Determination of such values can 
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be based on, at least in part, the monitored interaction history between a user and a 
particular database DBV Factors in this history which are considered in various 
embodiments of the present invention can include: how many times the user accesses 
database DB 1 ^ what search queries where used to identify database DB 1 ^ whether a 
purchase was made as a result of accessing database DB 1 ^ whether database DB* X was 
"bookmarked", as well as other factors which will become apparent to those skilled in the 
art from consideration of the present disclosure. A user can also provide an express value 
for database DB^. These factors are then combined to obtain a value, i.e. rating. In a 
preferred embodiment the factors are combined as a weighted sum. 
[0023] At present no optimal method for combining factors is believed to be 

known. However it should be understood that the present invention will be useful (in the 
sense that relatively highly valued sources will be more likely to be useful than randomly 
chosen sources) if any reasonable method is used to obtain the values ; e.g. any method 
which reflects that, other things being equal, that an information source demonstrates 
more value the more often it is accessed; that bookmarked sources demonstrate more 
value than sources which are not bookmarked; etc. Thus all such reasonable methods for 
combining factors to obtain values are within the contemplation of the present invention. 
[0024] Figures 2 A and 2B show a flow diagram of the operation of system 10 in 

accordance with the method of the present invention. At initial step 30 system 10 maps 
records in type 1 databases DB\ into predetermined formats for different types of 
options, and determines master weights MWj for corresponding type 2 databases DB j. 
Since databases DB 1 ! are maintained by various third parties they are mapped into 
predetermined formats so that ratings having different structures can be compared. For 
example similar dimensions of options may be described differently in different 
databases, e.g. reliability may be described in terms of mean time between failures, 
maintenance costs, durability, or expected useful life. Development of maps for various 
types of options and databases is well within the ability of those skilled in the art and 
need not be discussed further here for an understanding of the subject invention. Of 
course, in embodiments of the invention where databases are maintained to a uniform 
standard no mapping is necessary. Since, in the preferred embodiment described, 
database ratings DRj(i) are uniformly identified by indices i, j and expressed as an 
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increasing scalar value no mappings of type 2 databases DB j are required. Note, 
however, that ratings DRj(i) may differ for different types of options, e.g. database DB 2 j 
may rate database DB 1 ! differently for choosing among makes of cars and various 
hospitals. Master weights MWj are set by a system user to express the user's confidence 
in corresponding type 2 databases. At step 30 system 10 also gets parameter A which is a 
constant set by a user to reflect the importance that any chosen option of the selected type 
be rated as at least minimally satisfactory by all databases DB 1 !, as will be described 
further below. 

[0025] After initialization, at step 32 system 10 determines the type of options to 

be evaluated and at step 34 gets the relevant type 1 databases DB l i and associated 
mappings, minimum satisfactory ratings ORi(sat), and maximum ratings ORi(max), and 
relevant type 2 databases DB 2 j and associated master weights MWj, minimum 
satisfactory ratings DRj(sat), and maximum rating DRj(max). Relevant databases DB 1 ! 
are databases which include at least one option rating ORi(m,n) for a selected option m 
along a dimension of interest n. Relevant databases DB 2 j include at least one database 
rating DRj(i) of a relevant database DB 1 !. 

[0026] At step 36 all option values ORi(m,n) which are defined in relevant 

databases DB 1 !, for selected types of options m, along dimensions of interest are tested to 
determine if; 

ORi(m,n) < ORi(sat); 
and if so system 10 sets: 

(I) ORi(m,n) = ORi(sat) - A(ORi(max) - ORi(sat)) 
[0027] At step 40 index i is set equal to 1. 

[0028] At step 42, for all i,j, DRj(i) defined, system 10 sets: 

(2) Norm(DRj(i)) = (DRj(i) - DRj(sat))/(DRj(max) - DR/sat) 
[0029] At step 44. for all j, DRj(i) defined, system 10 sets weights Wi for 

databases DB 1 !: 

(3) Wi = Zj (MW j #Norm(DR j (i)))/(XjMWj); 
and continues to loop through steps 48 and 50 until all values for Wj are calculated. 
(Note that though weights Wi have been described for entire databases DB 1 ! distinct 
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weights along each dimension can be obtained without departing from the disclosure of 
the present invention by partitioning databases DB 1 * into separate databases along each 
dimension.) 

[0030] At step 52, shown in Figure 2B, index i is set equal to 1, and at step 56 for 

all i, Wi < 0, system 10 sets Wi = 0 to avoid a possible false choice based on the product 
of a negative Wi times a large Norm(ORi(m,n)). 
[0031] At step 58 all i,m,n ORi(m.n) defined, system 10 sets: 

(4) Norm (ORi(m.n)) = (ORi(m.n)- ORi(sat))/(ORi(max) - ORi(sat)) (Note that by 
(J) Norm (ORi(m,n)) = -A if ORi(m,n) < ORi(sat).) 

[0032] At step 64. for all i, ORi(m,n) defined, system 10 sets overall rating 

R(m,n) for option m along dimension n: 

(5) R(m,n) = (Wi»Norm(ORi(m ? n)))/(XiWi); 
and continues through inner loop steps 66 and 68 and outer loop steps 70 and 72 until all 
values for R(m,n) are calculated. 

[0033] At step 74 objective information of interest such as price is appended to 

ratings R(m,n) and at step 76 a list of ratings with any objective information is output and 
system 10 exits, 

[0034] Though various elements have been shown as physically distinct 

structures in Figure 1 for purposes of clarity, those skilled in the art will recognize that 
the functions of these elements can be carried out by a single system or the like. 
Particularly, though databases DBS and DB 2 j have been shown, and will be described 
below, as distinct structures for reasons of clarity, the information contained in these 
databases can be stored in a single database on a single storage device. Accordingly, 
reference herein to particular databases is not to be understood as being limited to distinct 
or separate data structures, but as used herein includes any form of data organization 
whereby the identified information items can be accessed as a group. 
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Example 

Table 1 

[A = 0.5] 

DB\: OR! (max) = 100, OR, (sat) = 0 



Optons/Dimensions 


Value 


Durability 


ServAVarranty 


Overall 
Quality 


Style 


Price(S) 


Brand #1 


69 


64 


0 


86 




78.99 


Brand #2 


74 


83 


0 


56 




34.99 


Brand #3 


78 


86 


70 


68 




67.99 


Brand #4 















DB^: OR 2 (max) = 10, OR 2 (sat) = 8 



Optons/Dimensions 


Value 


Durability 


ServAVarranty 


Overall 
Quality 


Style 


Price(S) 


Brand #1 










9 




Brand #2 














Brand #3 










-3 




Brand #4 














DBV OR 3 (max) = 1, OR 3 (sat) = 0 












Optons/Dimensions 


Value 


Durability 


ServAVarranty 


Overall 
Quality 


Style 


Price($) 


Brand #1 


-0.12 


0.21 


0 


0.79 




78.99 


Brand #2 


0.57 


0.45 


0 


0.07 




34.99 


Brand #3 














Brand #4 


0.78 


0.42 


0 


0.05 




19.99 


DB\ OR^max) = 100, OR^sat) = 


50 










Optons/Dimensions 


Value 


Durability 


ServAVarranty 


Overall 
Quality 


Style 


Price(S) 


Brand #1 


75 






86 


92 


63.00 


Brand #2 


97 






86 


85 


34.99 


Brand #3 


84 






88 


87 


19.99 


Brand #4 
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Table2 





MWj 


DRj(max) 


DRj(sat) 


DB\ DB l 2 


DB l 3 


DB^ 




DB2-1 


3 


100 


0 


56 78 


33 


91 




DB2-2 


10 


10 


5 


7.93 4.04 


1.46 


9.38 




DB2-3 


0.7 


1 


0 


0.52 


0.03 


— 










Table 3 

Overall Rating 








Optons/Dimensions 


Value 


Durability 


Serv/Warranty 


Overall 
Quality 


Style 


Price($) 


Brand #1 
Brand #2 




0.58(N=2) 
0.87(N=2) 


0.64(N=1) 
0.83(N=1) 


0.00(N=1) 
0.00(N=1) 


0.78(N=2) 
0.66(N=2) 


0.84(N=2) 
0.70(N=1) 


[63.00 - 
78 991 
34.99 


Brand #3 




0.72(N=2) 


0.86(N=1) 


0.70(N=1) 


0.73 (N=2) 


0.74(N=2) 


67.98 


Brand #4 




? 

■ 


? 

• 


? 

a 


? 

■ 


* 


? 

« 



[0035] A user wishes to choose among four brands of consumer products, i.e. 

options, which are rated by four type 1 databases DB\ through DB [ 4> as shown in Table 

OO 

1. The type 1 databases are rated by three type 2 databases DB i through DB 3 as shown 
in Table 2. From (2) and (3) and Table 2: 

Wl = (3(56 - 0)/(100 - 0)) + 10(7.93 -5)1(10 - 5)) + 0.7(0.52 - 0)/(l - 
0))/(3 + 10 + 0.7) = 

(1.68 + 5.86 + 0.36)/13.7 ~ 0.58 
W2 = (3(78 - 0)/(100 -0) + 10(4.04 -5)/(10 -5))/(l -0))/(3 + 10) = 

(2.34 - 1.92 + 0.021)/13 ~ 0.03 
W3 = (3(33 - 0)/(100 -0) + 10(1.46 - 5)/(10 -5) +0.7(0.03 - 0)/(l -0)) = 

(0.99 -7.08 + 0.021)/13.7 « -6.0, which, by step 56 in Fig. 2B, is 

set =0. 

W4 = (3(91 -0)/(100 - 0) + 10(9.38 - 5)/(10 - 5))/(3 + 13) = 
(2.73 + 8.76)/13 ~ 0.88 
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From (4) and (5) and Table 1 the overall rating for Brand 1 along the value dimension, 
R(U), is: 

R(l,l) = (0.58(69 -0)/(100 - 0) + 0(-0.5) + 0.88(75 - 50)/(100 -50))/(0.58 + 0 + 0.88) = 

(0.40 + 0 + 0.44)/(1.46) ~ 0.58 (Note that, by step 36 in Fig. 2A 
and step 58 in Fig. 2B, -A = -0.5 is substituted for the less than satisfactory option rating 
Norm(OR 3 (l,l)).) 

[0036] Similar results are obtained for other overall values and set forth in Table 

3, where N is the number of type 1 databases DB 1 ] contributing to the overall rating. 
Note no overall ratings where obtained for brand 4 since it was rated only by DB^ and 
weight W3 = 0. Also note that a range of objective information is reported for the brand 1 
price, reflecting the different information from different databases. 
[0037] Figure 3 shows a flow diagram of the operation of system 10 in 

accordance with an aspect of the present invention where master weights MWj are 
adjusted based upon a user's evaluation of a list of overall results. In a preferred 
embodiment, at step 80 system 10 sets j equal to 1. At step 82 a user inputs m' and n ? 
where m' is the option selected after considering the output list and n' is a critical 
dimension such as the dimension which the user considers as most important in making 
the choice. At step 84 system 10 calculates P(MWj'), where P(MWj') is the partial 
derivative of a function Fm',n'(MWj), which is defined as the deviation of the overall 
rating R(m',n') from the mean of the overall ratings R(m,n') along critical dimension n', 
as a function of all master ratings MWj, for all databases DB j for which R(m'n') is 
defined. System 10 then loops through steps 86 and 88 until all partial derivatives have 
been determined. At step 90 j is reset to 1 and at step 92 master weight MWj is adjusted 
by a factor, (1 + <xP(MWj)), where a is a small positive number for which an appropriate 
value can easily be determined by experimentation. System 10 then loops through steps 
94 and 96 until all values for MWj have been adjusted. 

[0038] In another embodiment of the invention Fm',n'(MWj), is defined as the 

deviation of the overall rating R(m',n') from the maximum of the overall ratings R(m,n') 
along critical dimension n 5 , as a function of all master ratings MWj, for all databases DB j 
for which R^'n') is defined and system 10 proceeds as shown in Figure 3. This 
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embodiment addresses the needs of a user who interested in how an option compares to 
the best of the other choices rather than to the average, Other techniques for adjusting 
master weights MWj will be readily apparent to those skilled in the control systems art 
and all such techniques are within the contemplation of the present invention. 
[0039] The embodiments described above and illustrated in the attached drawings 

have been given by way of example and illustration only. From the teachings of the 
present application those skilled in the art will readily recognize numerous other 
embodiments in accordance with the subject invention. Accordingly, limitations on the 
subject invention are to be found only in the claims set forth below. 
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